(19) 



J 



(12) 



(43) Date of publication: 

16.06.1999 Bulletin 1999/24 

(21) Application number: 98203417.5 

(22) Date of filing: 09.10.1998 



Europaisches Patentamt 
European Patent Office 
Office europ6endesbr vets (11) EP 0 923 225 A2 

EUROPEAN PATENT APPLICATION 

(51) Int. CI. 6 : H04N1/21 



(84) Designated Contracting States: 


(72) Inventor: 


AT BE CH CY DE DK ES FI FRGB GR IE IT LI LU 


Lathrop, George E., 


MCNLPTSE 


c/o Eastman Kodak Company 


Designated Extension States: 


Rochester, New York 14650-2201 (US) 


ALLTLVMKROSI 


(30) Priority: 20.10.1997 US 954367 


(74) Representative: 


Parent, Yves et al 


(71) Applicant: EASTMAN KODAK COMPANY 


KODAK INDUSTRIE, 


Dgpartement Brevets, 


Rochester, New York 14650 (US) 


CRT - Zone Industrielle 




71102 Chalon-sur-Sadne Cedex (FR) 



(54) Electronic camera using incremental image processing 

(57) An electronic still camera includes an image 
sensor for capturing an original image and generating 
image signals corresponding to the captured original 
image, an A/D converter for converting the image sig- 
nals into original digital image data representative of the 
captured original image, an image processor for incre- 
mentally processing the original digital image data to 
generate processed image data as each increment is 
processed and completed, and a non-volatile memory 
for storing the processed image data, wherein the non- 
volatile memory also temporarily stores the original dig- 
ital image data until the incremental processing is com- 
pleted. The incremental processing of the original 
image data can then be interrupted in order to capture a 
new image, while the processed image data for the 
completed increments of the original image is saved in 
the non-volatile memory. 
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ing detailed description of the preferred embodiments 
and appended claims, and by reference to the accom- 
panying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 5 
[0009] 

FIG. 1 is a block diagram of a digital still camera 
system configured according to the invention to 10 
process and store finished files in a removable stor- 
age device. 

FIG. 2 is a flow diagram describing the operation of 
the camera shown in Figure 1 in relation to incre- 
mental image processing performed according to 15 
the invention. 

FIG. 3 is a diagram of the image processing steps 
involved in the generation of finished files. 
FIG. 4 is a diagram of the Bayer color filter pattern 
used in connection with the image sensor in the 20 
camera shown in Figure 1 . 

DETAILED DESCRIPTION OF THE INVENTION 

[001 0] Because digital cameras employing electronic 25 
sensors and electronic processing and storage are well 
known, the present description will be directed in partic- 
ular to elements forming part of, or cooperating more 
directly with, apparatus in accordance with the present 
invention. Elements not specifically shown or described 30 
herein may be selected from those known in the art. 
Certain aspects of the embodiments to be described 
may be provided in software. Given the system as 
described in the following materials, all such software 
implementation needed to practice the invention is con- 35 
ventional and within the ordinary skill in such arts. 
[001 1 ] Figure 1 shows a block diagram of a digital still 
camera 10 capable of capturing and processing images 
irv accordance with the invention. The images are then 
delivered to a host computer 12. The digital still camera 40 
10 includes a conventional optical system 14 for direct- 
ing image light from a subject (not shown) to an image 
sensor 16. Although not shown as separate elements, 
the optical system 14 includes conventional lens optics 
for directing the image light through a diaphragm (which 45 
regulates the optical aperture) and a shutter (which reg- 
ulates exposure time). Alternatively, particularly for sen- 
sors that have a light protected readout region, the 
exposure time can be varied by electrically controlling 
the readout of the image sensor 16 in a conventional so 
manner. The image sensor 16, which includes a two- 
dimensional array of photosites corresponding to pic- 
ture elements of the image, is a conventional charge- 
coupled device (CCD) using, e.g., either well-known 
interline transfer, progressive scan, or frame transfer 55 
techniques. When the sensor 16 is exposed to image 
light to capture a particular image, analog image infor- 
mation corresponding to the particular image is gener- 



ated in respective photosites of the sensor 16. The 
sensor 16 includes a color filter array pattern, such as 
the Bayer color filter array pattern shown in Figure 4 and 
described in U.S. Patent No. 3,971 ,065, to provide color 
image data values. Accordingly, the analog image infor- 
mation output by the sensor 16 represents an uninter- 
polated stream of unique color values, one value for 
each pixel, corresponding to the color pattern of the 
color filter array. 

[001 2] The analog image signals are applied to an AJD 
converter 18, which generates digital image data from 
the analog input signals for each picture element. This 
digital image data is uninterpolated Bayer pattern image 
data in which each photosite provides a single red, 
green, or blue value corresponding to its position in the 
Bayer pattern. The uninterpolated digital image data is 
applied to a RAM buffer memory 20 having storage 
capacity for storing one image. A digital microprocessor 
22 (for example, a Motorola PowerPC MPC821 proces- 
sor) generally controls the exposure conditions by gen- 
erating, via programmable logic 24, the horizontal and 
vertical clocks needed for driving the sensor 16 and for 
clocking image information therefrom, and enables the 
AJD converter 18 in conjunction with the image buffer 20 
for each signal relating to a picture element. User inter- 
action with the microprocessor 22 is obtained through a 
set of user buttons 26, which include a shutter button for 
initiating an exposure sequence. 
[0013] The microprocessor 22 performs certain pre- 
liminary processing of the uninterpolated Bayer pattern 
digital image data, including blemish or defect conceal- 
ment and white balance, as well as the generation of a 
reduced resolution thumbnail image and a header for 
the image file according to a first file format. This prelim- 
inary processing is not particularly time intensive and 
does not significantly impact upon the capture timing of 
the camera. The preliminarily processed data is then 
transferred via an output interface 30 to a non-volatile 
storage device 32 and stored in the first file format. 
While not necessary for practice of the invention, the 
storage device 32 is a removable storage device, such 
as a memory card. This preliminarily processed digital 
image data in the storage device 32 is described herein 
as "raw" Bayer pattern image data in which the image 
data in the image file represents a single red, green, or 
blue value corresponding to its position in the Bayer pat- 
tern. 

[001 4] Once "raw" digital image data for the captured 
image has accumulated in the storage device 32, the 
stored image data is retrieved in incremental stages by 
the digital processor 22 and processed according to 
more comprehensive time-intensive image processing 
algorithms stored in an algorithm memory 28. These 
image processing algorithms provide, e.g., color filter 
array (CFA) interpolation, color correction, tone scale 
correction, edge enhancement, and compression. The 
processed image data is then transferred via the output 
interface 30 to the non-volatile storage device 32 in a 
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image data files. 

[0020] Because of the minimal processing required, 
the time between starting the sensor readout and com- 
pletely storing the Bayer pattern TIFF file in the storage 
device 32 is short (for example. 2 seconds). Once the 5 
"raw" image is stored in the storage device 32, the cam- 
era is ready to take another picture, as soon as the user 
again presses the shutter button in the user button sec- 
tion 26. However, if the user does not immediately press 
the shutter button, the microprocessor 22 in the camera 10 
begins to perform image processing of the stored "raw" 
Bayer pattern TIFF image, in order to create a "f inished" 
JPEG interchange format image file format as 
described in ISO/IEC 10918-1:1994. The microproces- 
sor 22 checks to see how many unprocessed TIFF is 
images exist (step 1 12), and begins to process the first 
unprocessed segment of the oldest TIFF image (step 
1 14). A segment may, for example, contain the data val- 
ues for a 1 6 row by 64 column area of the image. 
[0021 ] The oldest image is first checked to determine 20 
if it is partially processed (step 1 16). If the oldest image 
has no processed segments, the microprocessor 22 
begins to create a new JPEG file (step 1 18) by writing 
the JPEG header information, which includes the start 
of image and start of image JPEG segments, as well as 25 
an application segment, such as the Exit application 
segment described in "Digital Still Camera Image File 
Format Standard"(Exif), version 1.1, Japan Electronic 
Industry Development Association (JEIDA), May, 1997. 
This Exif application marker includes the TIFF thumb- 30 
nail data and other TIFF tags, for example the DateTime 
tag and the Fnumber tag. 

[0022] The microprocessor 22 then processes each 
image segment (step 122) using the processing steps 
shown in Fig 3. Exemplary steps include CFA interpola- 35 
tion 200, color correction 202, tone scale correction 
204, edge enhancement 206, and JPEG compression 
208. Typical CFA interpolation techniques for a Bayer 
pattern are described in U.S. Patent Nos. 5,506,619 and 
5,629,734. Examples of typical color correction, tone 40 
scale correction and edge enhancement are found in 
Digital Image Processing. 2nd ed., by William Pratt, 
Wiley, 1991 and Fundamentals of Digital Image 
Processing, by A.K. Jain, Prentice-Hall, 1989. The color 
image data may also be converted into JPEG standard 45 
color space (YCC CCIR 709 space) and downsampled 
to improve the compression ratio for chroma. A typical 
JPEG compression algorithm is the Discrete Cosine 
Transform (DCT). The processor begins by reading from 
the TIFF file the daita needed to process the next seg- so 
ment (step 120). This data may be slightly larger than 
the 16 x 64 pixel segment, since the values of "neigh- 
bor" pixels are needed to properly perform the color fil- 
ter array and edge enhancement processing. After the 
segment processing is completed, the microprocessor 55 
22 checks to see if this was the last segment of the 
image (step 124). If there are more segments, then data 
needed for the next segment is read from the TIFF file 



(step 120) and that segment is processed (step 122). 
[0023] The JPEG interchange format (JIF) file is com- 
plete after the last segment of the image has been proc- 
essed. This process may take approximately 20 
seconds per image, much longer than the time to cap- 
ture and store the "raw" TIFF file. After ail image seg- 
ments are fully processed, the TIFF file corresponding 
to the unprocessed image can be deleted (step 126). 
[0024] The purpose of saving each incremental step 
(i.e. saving the result of processing each segment) is to 
maintain earlier processing in case the user attempts to 
take another picture before the processing shown in Fig. 
3 is completed for the entire image (i.e. within 20 sec- 
onds of taking the last image). During any of the 
processing steps shown within the right-hand box 128 in 
Fig. 2, if the user wants to capture a new image, the 
shutter button in the user button section 26 is pressed, 
an interrupt section 130 is enabled and processing is 
returned to the capture sequence (steps 102-110). In 
particular, the processing of the current segment (step 
122) is halted (or alternately, aborted) and the user is 
allowed to capture a new image (step 102), which is 
immediately processed (according to steps 104-108) 
and stored on the removable storage device 32 (step 
110). Following the image capture(s) and storage, the 
previously halted incremental image processing step 
122 is resumed at the point of interruption (or the 
aborted segment is reprocessed). With this approach, 
the portion of the image processing that is lost is mini- 
mized to (at worst) the incremental step which was 
aborted and not to any previous incremental steps. This 
incremental image processing continues until the image 
is finished. 

[0025] While this example showed the incremental 
image processing performing the full set of algorithms 
on many (24 x 64 = 1536) image segments, it is also 
possible to completely process the full image sequen- 
tially using each algorithm shown in Fig. 3, and storing 
the intermediate results in the non-volatile memory, or 
by sequentially storing the output of each processing 
step for each segment in the non-volatile memory 32. In 
these two cases, after finishing a subsequent incremen- 
tal step, the microprocessor 22 would erase the results 
of the previous step. 

[0026] In summary, the invention is a camera architec- 
ture that uses the camera non-volatile transfer memory 
(such as a PCMCIA type III hard drive) to store image 
files as they are incrementally processed, and interrupts 
this processing whenever the user decides to take a 
new image. In effect, the camera processes images 
only during the idle times, i.e. times the camera isn't 
being used to capture an image. When the user wants 
to capture a new image, the camera "suspends" the 
image processing operations on the previously cap- 
tured "older" images, allows the user to capture "new" 
images, then resumes image processing of the older 
images, followed by the newer image. This allows the 
time between captured images to be minimized, without 
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each processed segment in the removable memory as 
the processing for each segment is completed. Or the 
method may incrementally perform a series of process- 
ing steps and store the results of each step in the 
removable memory as the processing for each step is 5 
completed, or incrementally processing the captured 
image in segments and further incrementally perform- 
ing a series of processing steps on each segment and 
storing the results of each step for each processed seg- 
ment in the removable memory as the processing for 10 
each step within a segment is completed. 
[0031] The invention has been described with refer- 
ence to a preferred embodiment. However, it will be 
appreciated that variations and modifications can be 
effected by a person of ordinary skill in the art without 15 
departing from the scope of the invention. 

PARTS LIST 

[0032] 20 

10 digital still camera 
12 host computer 
14 optical system 

16 image sensor 25 

18 A/D converter 

20 RAM buffer memory 

22 digital microprocessor 

24 programmable logic 

26 user button section 30 
28 algorithm memory 
30 interface 

32 removable, non-volatile storage device 
34 card reader 

36 processor 35 

38 keyboard 

40 monitor 

42 host interface 

44 cable 

46 interface 40 
Claims 

1. An electronic still camera for capturing an original 
image and performing image processing on the 45 
original image, said camera comprising: 

an image sensor for capturing the original 
image and generating image signals corre- 
sponding to the captured original image; so 
an A/D converter for converting the image sig- 
nals into original digital image data representa- 
tive of the captured original image; 
an image processor for processing the original 
digital image data to generate processed ss 
image data; and 

a non-volatile memory for storing the proc- 
essed image data, wherein the non-volatile 



memory also temporarily stores the original 
digital image data at least until the image proc- 
essor completes image processing of the origi- 
nal digital image data. 

2. A camera as claimed in claim 1 wherein the original 
image data is stored in a first file format and the 
processed image data is stored in a second file for- 
mat. 

3. A camera as claimed in claim 2 wherein the original 
image data is stored in a TIFF file format and the 
processed image data is stored in a JPEG file for- 
mat. 

4. A camera as claimed in claim 1 wherein the image 
processor can be interrupted in order to capture 
another image. 

5. A camera as claimed in claim 1 wherein the image 
processor implements incremental image process- 
ing such that the results of each increment are 
stored as each increment of processing is com- 
pleted. 

6. A camera as claimed in claim 5 wherein the incre- 
mental image processing processes the captured 
image in segments and stores the data for each 
processed segment in the non-volatile memory as 
the processing for each segment is completed. 

7. A camera as claimed in claim 6 wherein the proc- 
essed segments are stored in a JPEG interchange 
format bitstream separated by restart markers. 

8. A camera as claimed in claim 5 wherein the incre- 
mental image processing performs a series of 
processing steps and stores the results of each 
step in the non-volatile memory as the processing 
for each step is completed. 

9. A camera as claimed in claim 6 wherein the non- 
volatile memory is also removable. 

1 0- A method utilizing an electronic still camera for cap- 
turing an original image and performing image 
processing on the original image, said method com- 
prising the steps of: 

generating a capture signal in order to initiate 
an image capture of the original image; 
capturing the original image and generating 
image signals corresponding to the captured 
original image; 

converting the image signals into original digital 
image data representative of the captured orig- 
inal image; 

incrementally processing the original digital 
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(54) Electronic camera using incremental image processing 



(57) An electronic still camera includes an image 
sensor for capturing an original image and generating 
image signals corresponding to the captured original 
image, an A/D converter for converting the image sig- 
nals into original digital image data representative of the 
captured original image, an image processor for incre- 
mentally processing the original digital image data to 
generate processed image data as each increment is 
processed and completed, and a non-volatile memory 
for storing the processed image data, wherein the non- 
volatile memory also temporarily stores the original dig- 
ital image data until the incremental processing is com- 
pleted. The incremental processing of the original 
image data can then be interrupted in order to capture a 
new image, while the processed image data for the 
completed increments of the original image is saved in 
the non-volatile memory. 
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